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Effectiveness  of  Mailing  a 
CD  ROM  for  Navy  Recruiting 

Peggy  A.  Golfin 
Michael  Y.  Katz 


In  June  1998,  CNA  completed  a  CD  ROM  for  Navy  Recruiting  Command 
(CNRC),  which  CNRC  subsequently  titled  “Let  the  Journey  Begin.” 

In  January  1999,  CNRC  asked  CNA  to  conduct  an  experiment  that  involved 
mailing  2,000  of  these  CDs  to  randomly  selected  students  of  community 
colleges  and  vocational  institutions  and  following  up  to  determine  whether  any 
recipients  were  recruited.  This  annotated  briefing  reports  on  this  experiment. 


* 


1 
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Background 

•  CNA  created  the  “Let  the  Journey  Begin” 
CD  ROM  in  1998 

•  Purpose  was  to  assist  recruiters  on 
community  college  campuses 

•  ASN  suggested  a  mailing  of  CD  ROMs 


In  1998,  as  part  of  a  self-initiated  study,  CNA  created  a  CD  ROM  for  CNRC.  It  was 
intended  (a)  to  show  how  the  medium  could  be  used  to  assist  recruiters  and  (b)  to  serve  as 
a  prototype  for  future  efforts.  We  developed  the  CD  for  use  in  enlisted  recruiting  on 
community  college  campuses,  particularly  at  job  fairs  and  in  the  college  unions.  See  [1] 
for  more  details. 

This  CD  was  one  of  only  a  few  available  to  enlisted  recruiters  at  that  time.  CNRC  decided 
to  use  the  prototype  CD  and  make  it  available  for  all  of  enlisted  recruiting — not  just  for  the 
community  college  market.  In  addition,  the  Assistant  Secretary  of  the  Navy  for  Manpower 
and  Reserve  Affairs  (ASN  (M&RA))  suggested  that  CNRC  experiment  with  mailing  the 
CD  to  a  random  sample  of  students  to  determine  its  cost-effectiveness  in  generating 
contracts.  All  of  the  services  spend  large  sums  of  money  each  year  mailing  form  letters 
and  brochures  to  targeted  audiences,  such  as  high  school  seniors.  The  cost-effectiveness 
of  these  efforts  is  well  known;  it  often  takes  five  to  six  separate  mailings  to  prompt  a 
response.  The  cost  of  reproducing  a  CD  is  about  $1.00  each,  and  it  costs  at  least  that  much 
to  produce  a  letter  and  enclosures  for  a  mailing.  The  CD,  therefore,  had  the  potential  to  be 
at  least  as  cost-effective  as  traditional  mailing  methods  of  recruiting. 

By  targeting  a  mailing  of  the  CD  ROM  to  community  college  students,  CNRC  would 
eliminate  the  possibility  that  the  recipient  could  not  view  the  material  because  virtually  all 
college  students  have  access  to  computers  on  campus. 
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Experiment  Parameters 

•  CNRC  requested  name  lists  from 
community  colleges  from  several  districts 

•  2,000  community  college  and  vocational 
institution  students  were  randomly  chosen 

•  Selection  criteria: 

-  Address 

-  Age 

-  75%  male 


To  track  the  outcome  of  the  mailing,  we  provided  a  single  point  of  contact  (POC)  on  a 
cardboard  sleeve  for  the  CD  ROM.  The  POC  information  included  the  name,  e-mail,  and 
toll-free  number  of  a  member  of  an  internet  recruiting  team,  known  as  the  “cyberspace 
team,”  located  at  CNRC  headquarters  in  Arlington,  Virginia.  (For  more  information  about 
the  Navy’s  cyberspace  team,  see  [2].)  CNRC  made  plans  in  the  summer  of  1998  to  make 
36,000  copies  of  the  CD  ROM  for  distribution  to  each  high  school  (about  26,000), 
community  college  (1,400),  and  enlisted  recruiter  (4,500)  plus  2,000  as  a  fulfillment  item, 
and  2,000  for  the  mailing  experiment.  The  copies  were  available  in  mid-November. 

During  the  summer  of  1998,  CNRC  requested  community  college  and  vocational  institution 
name  lists  from  Education  Specialists  (EdSpecs)  in  four  Navy  Recruiting  Districts  (NRDs). 

It  received  over  30,000  names  from  which  to  draw  the  sample  (see  appendix  A  for  lists  of 
the  NRDs  and  institutions).  Each  institution  varied  by  the  amount  of  information  it  included 
for  each  student.  All  except  one  provided  both  a  name  and  an  address,  but,  in  some  cases, 
that  is  all  that  was  provided.  Other  institutions  provided  more  complete  information,  such  as 
age,  social  security  number  (SSN),  major,  GPA,  and  phone  number. 

In  January  1999,  CNRC  asked  CNA  to  conduct  the  experiment.  We  randomly  selected  2,000 
names  from  the  original  list,  after  first  censoring  observations  that  did  not  contain  complete 
address  information  and  deleting  people  we  could  determine  were  over  the  age  of  27.* 


*We  chose  27  as  an  arbitrary  cutoff,  but  the  propensity  to  enlist  is  a  decreasing  function  of  age,  and  the  vast 
majority  of  enlisted  recruits  are  under  this  age. 
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Experience  has  shown  that  those  with  an  Associate  degree  score  higher  on  the 
Armed  Forces  Qualification  Test  (AFQT)  and,  therefore,  qualify  at  a  higher  rate 
for  critical  ratings,  such  as  the  nuclear  field  and  the  Advanced  Electronics/ 
Computing  Field  (see  [3]).  Because  these  fields  have  a  greater  number  of 
requirements  for  men  than  for  women,  we  also  weighted  the  sample  to  contain 
about  75  percent  men.* 


*We  could  not  specify  the  number  of  men  precisely  because  gender  was  not  indicated  on  every 
record. 
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Timeline 

•  Mailed  1,936  CDs  on  5  February  1999 

•  Mailed  200  followup  letters  on  10  February 

•  Throughout  February  and  March,  300  CDs 
were  returned  as  undeliverable 

•  An  additional  236  CDs  were  recycled  and 
mailed  to  new  names 

•  Total  of  1,872  CDs  were  mailed  and 
presumably  received 


On  5  February  1999  (a  Friday),  we  mailed  1,936*  CDs,  first  class,  using  the  standard  33-cent 
postage  rate.  The  following  Monday,  approximately  30  of  the  CDs  were  returned  from  the 
central  post  office  with  postage  due  (1 1  cents  each)  because  the  CD  sleeve  does  not  conform 
to  the  standard  size  for  the  33-cent  rate.  We  tried  to  determine  why  so  few  CDs  were 
returned  for  additional  postage.  The  manager  of  the  local  post  office  informed  us  that 
different  clerks  processed  the  entire  set,  and  apparently  one  clerk  caught  the  mistake.  The 
manager  assured  us  that  such  a  large  mailing  would  not  be  delivered  with  postage  due,  that 
the  other  clerks  most  likely  did  not  notice  the  need  for  additional  postage,  and  that  the 
remaining  CDs  were  probably  delivered.  In  any  event,  we  decided  to  send  a  followup  letter 
to  a  random  subsample  of  200  of  the  original  1,936,  explaining  the  purpose  of  the  CD,  noting 
again  the  name  and  contact  information  of  the  POC,  and  apologizing  in  case  they  received  the 
CD  with  postage  due.  We  also  thought  that  a  followup  letter  provided  an  additional  aspect  of 
the  experiment— whether  a  letter  of  explanation  and  a  reminder  of  the  POC  would  increase 
the  response  rate. 

During  the  next  several  weeks,  300  CDs  came  back  to  CNA  as  undeliverable.  Many  of  these 
also  indicated  that  additional  postage  was  due.  Apparently,  some  of  the  clerks  at  the  sending 
post  office  or  clerks  at  the  receiving  office  noticed  that  the  CD  was  not  a  standard  size.  It  is 
impossible  to  determine  what  percentage  of  the  total  CDs  were  flagged  in  this  way. 

CDs  that  were  returned  in  good  condition  were  mailed  out  again.  In  total,  1,872  CDs  were 
ultimately  delivered. 


♦This  was  the  total  number  of  CDs  available  to  us  for  mailing. 
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Outcome 


•  Contracts 

-  Method  for  matching 

-  20  confirmed,  2  more  potential 

•  Conversion  rate  for  CDs  and  letters 


The  most  important  measure  of  the  effectiveness  of  the  experiment,  at  least  in  the  long 
term,  is  how  many  recipients  were  ultimately  recruited.  We  have  compared  the  names  of 
those  who  received  a  CD  to  names  of  new  contracts  in  the  Navy’s  enlisted  reservation 
database  (PRIDE)  for  February  through  August.  If  the  SSN  of  each  recipient  were 
known,  we  would  be  able  to  say  with  certainty  how  many  contracts  resulted.  However, 
we  had  the  SSNs  of  only  a  relatively  small  number  in  the  sample.  For  those  for  whom 
we  did  not  have  SSNs,  we  attempted  to  match  on  the  following  criteria:  last  name  and 
first  name  must  match  completely,  middle  name  or  middle  initial  must  not  be  in  conflict 
(in  other  words,  in  PRIDE  the  middle  name  may  be  John,  and  in  the  sample  the  middle 
name  is  simply  listed  as  J,  or  no  letter  is  provided),  and  age  must  be  19  or  older. 

We  could  have  narrowed  the  criteria  further  by  requiring  a  match  in  the  state  or  by 
requiring  the  person  to  have  at  least  12  years  of  school  and  a  high  school  degree.  We 
decided  against  this,  however,  when  the  cyberspace  recruiters  called  five  or  six 
recipients  and  learned  that  some  had  already  graduated  from  the  2-year  college  and  were 
either  attending  a  4-year  college  in  another  state  or  had  found  a  job  in  another  location. 

In  addition,  we  learned  that  one  of  the  people  who  received  the  CD  was  a  high  school 
dropout  who  had  also  dropped  out  of  college.  This  person  was  recruited,  but  had  been 
working  with  a  recruiter  for  about  a  year.  Consequently,  we  decided  to  check  all 
accessions  that  matched  on  just  the  simple  criteria  outlined. 
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We  requested  that  the  cyberspace  recruiters  follow  up  with  each  potential  CD  ROM 
contract  with  a  phone  call.  If  no  phone  number  was  available,  we  asked  that  they  call 
the  recruiting  station.  Using  this  method,  they  were  able  to  confirm  20  contracts,  and 
two  other  potential  recruits  (see  appendix  B  for  a  complete  list  of  names  and  station 
IDs  of  the  20  confirmed  and  2  potential  recruits).  For  most  of  the  20  confirmed 
contracts,  the  cyberspace  recruiters  could  only  contact  the  recruiter  because  most  of 
the  recruits  had  already  shipped  to  boot  camp.  In  each  of  these  cases,  the  recruiter  had 
knowledge  that  the  recruit  had  received  the  CD  ROM  because  he  or  she  had 
mentioned  it  to  the  recruiter.  The  two  potential  recruits  matched  in  name,  had  already 
shipped  to  boot  camp,  but  had  not  mentioned  the  CD  ROM  to  their  recruiters. 
However,  the  recruiters  for  these  two  contracts  identified  them  as  coming  from  a 
national  lead  that  had  been  sent  in  either  March  or  May — after  the  CD  ROM  had  been 
sent. 

Assuming  that  each  recruit  and  recruiter  understood  that  we  were  inquiring  about  the 
“Let  the  Journey  Begin”  CD,  the  conversion  rate  of  CDs  to  contracts  is  1  percent.  The 
follow-up  letter  did  not  increase  this  rate.  Of  the  20  known  contracts,  only  1  received 
the  letter.  This  is  a  .5-percent  conversion  rate  (1  divided  by  200),  compared  to  the 
1 -percent  rate  (19  divided  by  1,672)  of  those  who  did  not  receive  the  letter. 
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Cost-Effectiveness 

•  Cost  of  each  contract  for  CD:  $650 

•  Cost  of  contract  from  workforce  mailing  for 
FY  1996:  $6,492 

•  Average  cost  from  all  national  leads  sources 
for  FY  1996:  $1,232 


Is  mailing  a  recruiting  CD  ROM  a  cost-effective  method  for  generating  enlisted 
contracts?  We  believe  that  our  results  indicate  that  it  is.  Consider  the  following.  It 
costs  about  $1.30  to  reproduce  and  mail  a  CD  ROM.  Adding  roughly  $10,000  in 
labor  costs  to  gather  name  lists  and  to  prepare  and  mail  the  CDs,  as  well  as  the 
followup  letter  to  the  subsample  of  200,  the  total  cost  of  this  leads-generating 
activity  was  about  $13,000.  This  translates  to  a  cost  per  enlisted  contract  for  this 
experiment  of  $650.  For  comparison,  in  FY  1996,  the  cost  per  enlisted  contract 
from  direct  mail  efforts  to  the  workforce*  was  $6,492  (according  to  the  Cost  Per 
Lead  Report,  FY  1996  Funding,  prepared  13  April  1998).  Using  this  as  a 
benchmark,  the  contracts  from  this  effort  have  cost  about  one-tenth  of  other  direct 
mail  efforts.  In  fact,  these  contracts  cost  less  than  the  average  cost  of  all  national 
leads-generated  enlisted  contracts,  which  in  FY  1996  was  $1,232.** 


♦Community  college  students  are  in  the  workforce,  which  we  define  as  anyone  in  the  recruitable 
population  who  is  not  in  high  school — including  both  high  school  graduates  and  dropouts. 

**Some  of  these  recruits  might  have  joined  the  Navy  even  if  they  had  never  received  the  CD  ROM, 
but  this  is  something  that  we  can’t  measure.  However,  the  same  is  true  of  any  CNRC  leads- 
generating  activity.  The  typical  method  of  evaluating  the  cost-effectiveness  of  all  of  these  types  of 
activities  is  simply  to  count  all  recruits  as  being  derived  from  that  particular  effort.  By  following  the 
same  methodology,  our  cost  comparison  to  other  leads-generating  efforts  is  appropriate. 
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Of  course,  other  contracts  could  have  resulted  from  this  mailing,  but  we  are  not 
able  to  identify  more.  For  instance,  a  CD  ROM  could  be  passed  around  to 
acquaintances  of  the  recipient  more  frequently  than  a  form  letter  sent  from  the 
Navy.  Under  such  circumstances,  it  would  not  be  possible  to  identify  any  of 
these  types  of  recruits.  We  also  might  have  missed  recruits  resulting  from  the 
CD  ROM  if  the  name  that  was  provided  to  us  by  the  colleges  was  not  the  official 
name  of  the  student.  In  other  words,  if  the  college  gave  us  the  name  of  “Skip 
Thomas,”  but  his  real  name  was  Robert  Thomas,  we  would  not  have  been  able 
to  identify  Robert  as  a  contract  resulting  from  the  CD  ROM. 
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Lessons  Learned 

•  Packaging 

•  Postage 

•  POC 

•  Nature  of  name  lists 


Even  though  this  experiment  has  proved  to  be  cost-effective,  we  believe  it  could  have  been 
even  more  successful,  for  a  variety  of  reasons.  Most  of  these  reasons  are  valuable  lessons  for 
future  endeavors;  some  are  one-time  difficulties.  We  will  describe  each. 

First,  we  learned  that  the  packaging  of  the  CD  ROM  was  not  optimal.  We  printed  address 
labels  and  attached  them  to  the  cellophane  wrapper  around  the  cardboard  sleeve.  However, 
when  some  of  the  CDs  were  returned,  we  noted  that  the  ink  used  to  cancel  the  stamp  smeared 
on  the  cellophane,  often  bleeding  into  the  address  or  otherwise  making  the  CD  very  messy  to 
handle.  To  recycle  the  CD,  we  needed  to  peel  the  label  and  the  stamps  off,  which  often 
destroyed  the  cellophane  wrap,  making  it  impossible  to  reuse  the  CD.  We  recommend  that 
future  mailings  include  a  postage  permit  on  the  cardboard  sleeve  itself,  so  that  no  postage  or 
cancellation  is  required.  A  postage  permit  would  also  have  eliminated  the  problem  of  not 
using  the  correct  postage. 

The  second  major  difficulty  we  encountered  was  the  result  of  a  number  of  concurrent  events. 
When  CNRC  created  the  sleeve  in  June  1998,  they  envisioned  that  the  experiment  could 
begin  the  following  September  or  October.  CNRC  was  moving  the  location  of  the 
headquarters  from  Arlington,  VA,  to  Millington,  TN,  from  the  fall  of  1998  through  June 
1999.  However,  they  knew  that  the  cyberspace  recruiters  were  not  scheduled  to  move  until 
the  end  of  that  time  frame.  Thus,  they  believed  that  using  a  name,  phone  number,  and  e-mail 
account  for  one  of  these  recruiters  would  not  present  any  difficulties.  However,  the  delivery 
of  the  CDs  was  delayed  until  November,  and  CNRC  decided  that  mailing  the  CDs  at  that 
time  was  not  optimal  because  college  students  would  be  concentrating  on  finals  and  their 
holiday  break. 
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Thus,  CNRC  decided  to  postpone  the  experiment  until  the  early  part  of  the  spring 
semester.  By  that  time,  die  recruiter  who  had  been  appointed  as  the  POC  for  the  CD 
ROM  had  been  reassigned,  and  the  remaining  recruiter  was  on  leave  for  a  large  portion 
of  March  and  April.  Personnel  were  reassigned  to  the  cyberspace  effort  on  a 
temporary,  ad  hoc  basis,  but  they  were  not  aware  of  the  ongoing  experiment.  We 
believe  that  some  recipients  of  the  CD  may  have  called  the  toll-free  number  and  asked 
for  the  POC  by  name.  When  they  were  informed  that  he  no  longer  worked  there,  they 
may  have  hung  up.  We  also  know  of  one  recipient  who  e-mailed  the  POC  in  response 
to  the  CD  ROM,  but  that  e-mail  was  mistakenly  deleted  before  the  person  could  be 
contacted.  By  mid-May,  there  was  no  longer  anyone  at  CNRC  to  answer  the  toll-free 
number  printed  on  the  CD,  and  the  e-mail  address  was  no  longer  valid.  While  these 
difficulties  with  the  cyberspace  team  have  now  been  resolved,  we  do  not  believe  that 
we  can  accurately  assess  how  many  recipients  actually  tried  to  contact  the  POC 
identified  on  the  CD  sleeve,  nor  can  we  determine  whether  others  might  have  been 
recruited  had  these  problems  not  occurred. 


A  final  concern  with  the  name  lists  is  the  accuracy  and  timeliness  of  their  use.  We 
believe  that  many  of  the  CDs  would  not  have  been  returned  had  the  lists  been 
requested  closer  to  the  time  of  the  mailing.  We  did  not  have  information  concerning 
the  college  class  of  the  student,  but  certainly  the  lists  contained  the  names  of  students 
who  graduated  last  May  (recall  that  the  lists  were  requested  in  the  late  spring  and  early 
summer  of  1998).  Some  of  these  people  either  went  to  a  4-year  college  (and  would 
therefore  have  benefited  more  by  information  for  officer  programs)  or  were  employed 
and  had  moved  out  of  their  parents’  houses.  From  the  feedback  we  received  from  the 
few  recipients  who  were  called,  in  both  of  these  types  of  cases,  the  parents  had  not 
forwarded  the  CD  to  their  child  or  returned  it  to  us.  Thus,  we  cannot  accurately 
determine  how  many  of  the  CDs  actually  reached  the  intended  recipients. 
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Recommendations 

•  Incorporate  the  CD  ROM  in  national  leads 
plan 

-  More  copies  need  to  be  made 

-  Timing  is  important 

•  Experiment  with  mailing  in  October  and 
February 

•  Create  more  CD  ROMs 


The  CD  ROM  has  proved  to  be  a  cost-effective  method  of  generating  enlisted 
contracts  (and  perhaps  even  officer  contracts).  We  recommend  that  CNRC  now  make  a 
mailing  of  the  CD  part  of  the  annual  national  leads  plan,  incorporating  lessons  learned 
from  this  experiment.  In  particular,  we  believe  that  each  CD  should  provide  the  new 
cyberspace  POC  information  (using  generic  title  of  “Career  Counselor”  instead  of  a 
person’s  name).  Headquarters  has  now  moved,  and  eight  recruiters  are  assigned  to  this 
effort,  so  the  cyberspace  difficulties  described  previously  should  no  longer  be  a 
problem. 

A  postage  permit  should  be  incorporated  into  the  new  sleeve,  and  it  might  be 
beneficial  to  enclose  a  letter  with  each  CD  explaining  what  the  CD  contains,  why  it  is 
being  sent,  how  to  run  it,  and  how  to  reach  the  cyberspace  POC.  While  the  letter  has 
not  proved  to  be  more  effective,  our  experiment  sent  it  as  a  separate  mailing.  It  would 
not  cost  much  more  to  enclose  this  information  on  a  small  card  inserted  into  the  CD. 
For  tracking  purposes,  a  separate  e-mail  account  could  be  established  for  this  effort, 
such  as  CD@cnrc.navy.mil. 

We  also  believe  that  the  community  college  market  is  a  good  one  to  receive  the  CD 
because  we  know  that  these  students  have  access  to  computers.  However,  the  next 
attempts  might  also  include  some  high  school  seniors  to  determine  the  cost- 
effectiveness  of  that  market.  February  is  a  good  time  of  year.  This  is  the  time  when 
graduating  students  put  the  most  effort  into  their  job  search.  Many  community  college 
students  graduate  in  December,  however,  so  a  smaller  mailing  could  be  done  in 
October  to  students  graduating  in  December. 
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Given  the  time  required  to  create  a  new  sleeve  and  reproduce  the  CD,  it  is  not 
possible  to  do  a  mailing  in  October  1999. 

Ultimately,  the  CD  may  be  useful  for  local  Navy  Recruiting  Districts  as  a  leads- 
generating  tool,  particularly  because  community  college  name  lists  are  typically 
gathered  at  the  district  level.  For  their  use,  however,  they  would  most  likely 
want  to  put  the  NRD  leads  personnel  as  the  POC.  This  would  require  separate 
sleeves  for  each  NRD,  and  perhaps  each  NRD  would  then  be  responsible  for 
reproducing  the  CD  to  meet  their  own  particular  needs. 

Finally,  we  recommend  that  CNRC  consider  creating  additional  CDs,  building 
on  the  feedback  from  “Let  the  Journey  Begin.”  We  originally  included 
information  on  just  four  recruiting  fields;  nuclear,  advanced  electronics/ 
computing,  advanced  technical,  and  medical.  If  CNRC  begins  to  recruit  more  to 
occupations,  perhaps  a  new  CD  ROM  dedicated  to  all  of  the  fields  would  be 
more  useful.  Also,  a  CD  ROM  targeted  to  the  4-year  college  market  may  also 
prove  to  be  cost-effective. 


Appendix  A:  List  of  colleges  and  vocational  institutes 


NRD 

Buffalo,  NY 


Jacksonville,  FL 


St.  Louis,  MO 


Los  Angeles,  CA 


Name  of  Institution 
Briarwood  College 
Bryant  and  Stratton 

Houstaonic  Community-Technical  College 
Mater  Dei  College 
Orange  County  Community  College 
SUNY 

Bainbridge  College 

Central  Florida  Community  College 

Chipola  Junior  College 

Arkansas  State  University 
Belleville  Area  College 
East  Central  College 
Forest 

John  A.  Logan  College 
Shelby  State  Community  College 
Southeastern  Illinois  College 

Ventura  College 
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Appendix  B:  Names  and  station  IDs  of  CD  ROM 
recipients  who  have  contracted  with  the  Navy 


Nam? 

Jason  Ash 
Michael  Bartlett 
Greg  Brown 
Jason  Burt 
Shawn  Clark 
Jason  Davis 
Jason  Hill 
James  Jones 
David  Kline 
Jason  Miller 
Mathew  Miller 
Angel  Perez 
David  Pierce 
Jason  Shelton 
Eric  Smith 
Michael  Smith 
Anthony  Taylor 
Christopher  Williams 
Michael  Wilson 
Jason  Young 


Station  ID 

531230 

521532 

315020 

840560 

840530 

830490 

838400 

120418 

846174 

312160 

542120 

840200 

527600 

839321 

547260 

103403 

527601 

314080 

531070 

547248 


Potential  recruits 

Michael  McDonald  838300 

Joseph  Smith  547035 
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